Aiming at the problem that traditional single factor methods cannot make full use of the relevant information of time series and has the poor accuracy and reliability of time series prediction， a time series prediction model based on multimodal information fusion，namely Skip-Fusion， was proposed to fuse the text data and numerical data in multimodal data. Firstly， different types of text data were encoded by pre-trained Bidirectional Encoder Representations from Transformers （BERT） model and one-hot encoding. Then， the single vector representation of the multi-text feature fusion was obtained by using the pre-trained model based on global attention mechanism. After that， the obtained single vector representation was aligned with the numerical data in time order. Finally， the fusion of text and numerical features was realized through Temporal Convolutional Network （TCN） model， and the shallow and deep features of multimodal data were fused again through skip connection. Experiments were carried out on the dataset of stock price series， Skip-Fusion model obtains the results of 0.492 and 0.930 on the Root Mean Square Error （RMSE） and daily Return （R） respectively， which are better than the results of the existing single-modal and multimodal fusion models. Experimental results show that Skip-Fusion model obtains the goodness of fit of 0.955 on the R-squared， indicating that Skip-Fusion model can effectively carry out multimodal information fusion and has high accuracy and reliability of prediction.